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A ESTR ACT 

The paper is addressed to the problem ct how to 
accomplish evaluation ot pupil Personnel Services (PPS) and its 
product, i.e. youth. It dist ingu ishes between direct and indirect 
evaluation ot the product ai:d the process. Aspects ot PPS which make 
evaluation ditticult, but not impossible are noted: (1) the provision 

of PPS is partially an art and its speci t ica t ion can only be 
approached; |2) the product as a dynamic developing person and 
control and observation are properly limited; and (3) goals are 
difficult to operationalize. Ar. evaluation paradigm which can be 
followed by PPS prof essicrals is described and applied, by way of 
illustration, tc a dropout prevention program. (TL) 
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Till,: Solving the Mystery of Pupil Personnel service. Evaluation 

Abstract: 

Evaluation of pupil personnel services (Pis) has hecoue a «nJor 
concern of the guidance profession. Vet, guidance literature reveal, 
n ttle .tout the concept. This paper ccplenent. Striven-, work (1967) 



in reducing the wr.tery of evaluation. It distinguishes b.t.een direct 
and indirect evnluatlon. note, the aspects of PPS which nek. evaluation 
difficult, tut not inpossltle, and d.scrites an evaluation puradig. 
that can te followed ty a PPS professional. The application of the 
pared 1 go is then illustrated for a dropout prevention progran. 
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SOLVING THE MYSTERY OF IPS EVALUATION 



CHARLES C. HEALY 
SCHOOL OF EDUCATION 
UNIVE R SITY OF C ALIF ORNIA, LOS ANGELE S 

A new term has entered the PPS dictionary, and like many new terms 
it is shrouded In mystery. The term is evaluation. Although new to the 
PPS dictionary, the concept has been known to man from the beginning of 
language. This paper will clarify evaluations maaning and application 
to PPS. 

Ve all know what evaluation is -- the process of Judging whether a 
procedure and its product meet specific criteria. Scriven (1567) has 
distinguished product evaluation, which he termed sutunativo, and procc&s 
evaluation, which he termed formative. However, what is new to PPS is 
not this important distinction, but the growing demand that somehow PPS 
end its product, our youth, be evaluated. And of course the mystery is 
how to accomplish such evaluation. 

Direct and Indirect Evaluation 

Examination of the concept of evaluation quickly suggests two ap- 
proaches. We can examine the product and process by directly observing 
and measuring their consistency with our criteria, or ve can examine cor* 
relates or ccncornmitAtits of our criteria, thereby Indirectly evaluating 
process and product. For example, direct evaluation of the PPS process 
would Involve determining whether a student received the services pre- 
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scribed by the PPS plan; that is, were his diagnosed needs the basis of 
his assignment to the selected services, and did those services operate 
as programmed? Likewise, direct evaluation o£ the prod uct would require 
us to investigate whether our graduate had achieved his PPS objectives; 
that is, could he now state a career plan? Did he apply his decision- 
making strategy to career decisions? In contrast, indirect evaluation of 
PPS proces3 would not involve us in looking at the PPS process directly; 
instead we would examine the concorani tants of an effective process. Is 
there an adequate counselor-s tudent ratio, and are the counselors creden- 
tialod, etc.? Likewise, in I ndirect evaluation of _ t h e PPS product , ve 
would examine ths correlates of our criteria -- How many graduates remem- 
ber their counseling? What percent of graduates arc satisfied with their 
Job? What percent have changed Jobs? 

r jOth approaches are complementary and should be used in concert. In- 
direct evaluation allows us to quickly Judge whether our criteria are at- 
tainable with the existing procedures. Since indirect evaluation allows 
only a "no" or "maybe" judgment of ef k’ec tivereso, direct evaluation is neces 
fary to clarify the " may be." 

Traditionally, however, evaluation of PPS has been limited to indi- 
rect evaluation. This cay have happened because adequate resources for 
effective PPS were so infrequently available that when a program’s re- 
sources were found to be adequate, it was considered wasteful to evaluate 
further. More urgent was the need to stimulate the programs to obtain 
adequate resources. However, PPS resources have tripled in the last dec- 
ade; and now it is urgent to pursue direct evaluation of PPS after indi- 
rect evaluation indicates a "maybe." 
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Both direct and indirect evaluation require the specification of 
criteria; that is, the evaluator must define standards against which he 
will Judge his process or product. Direct evaluation requires that stan- 
dards be defined by measurable behaviors. Indirect evaluation needs only 
general descriptive standards, since the correlates of those standards 
vil.1 be scrutinized, not the standards themselves. 

After specifying his standards and gathering his feedback, the eval- 
uator judges whether his standards have been achieved. If they were, his 
Job is complete. If they were not achieved, he modifies the operation in 
accordance with feedback and collects new feedback, modifying and recycl- 
ing until the standards are achieved. 

Judgments about .he standards achieved by a PPS program should not 
be equated with Judgments of worth. Judgments of worth are based on the 
degree of goal achievement and comparisons of the evaluated program with 
comparable programs. Therefore, knowledge of a program’s effectiveness in 
achieving its goals provides only part of the dsta needed for Judging its 
worth. 

Difficulties of Evaluation 

Specification of process and product in PPS have been difficult and 
controversial tasks. Because provision of pps is partially an art, its 
specification can only be approached; because the product is a dynamic, 
developing person,, control and observation of him are properly limited. 
Vhcn PPS researchers attempt evaluation, however, controversy compounds 
these difficulties. PPS professionals disagree about the importance of 
different process parameters and g"»al behaviors and the possibility of 
operationalizing goals. Tne difficultyof operationalizing goals is pax- 
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ticularly vexing because inadequate definitions of a goal in observable 
terms prevent development of the goal. More serious, however, is omitting 
some goals because they cannot be operationalized easily. Such omissions 
lead to development of only oart of the person. Since the superficial is 
easier to operationalize, failure to try to operationalize all goals can 
misdirect PPS into attempting only the superficial. 

The multifaceted nature of PPS inakes evaluation even more difficult. 
Ideally, PPS integrates distinct services which serve different needs of 
different students. Each student receives more or less individualized 
services, according to his need. Services overlap in varying degrees and 
it becomes difficult to identify which students received which services. 
Goals of services also overlap and may sometimes even conflict, further 
inc teasing the evaluation problem, while emphasiz ing the need for evalua- 
tion. 

In spite of the real difficulties inherent in specifying process and 
goal parameters, especially in light of its many facets, evaluation of 
VVS must be undertoken by IPS professionals. Upgrading of the PPS pro- 
cess and product requires evaluation and more end more school boards are 
demanding it to justify expansion and, of more concern, to Justify contin- 
uation of PPS. If the PPS profession fails to evaluate Its services, 
others, who are less qualified, are ready to do so. 

Steps In Eva l uation 

I shall not try to resolve the controversies of process and outcome 
parameters, but rathei I shall describe procedures to be followed in mak- 
ing an evaluation, once tentative parameters are accepted. Although spec- 
ifying criteria and choosing measures of those criteria are difficult tasks, 
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they are only the first step in the process. I shall describe the rest 
of the process in the hope that the PPS professional will realize that he 
can, and must, actively be engaged in evaluation. 

Evaluation of a PPS program starts by specifying goals in measurable 
*enns and agreeing on appropriate measures. If evaluation is continuous, 
che first step, although difficult and controversial, need not be insur- 
mountable, for goals and measures can be deleted, modified, and substi- 
tuted as feedback is received. 

Step 2 consists in arranging for measurement of the goal behaviors. 
Questions concerning who, when, and v'here need to be answered ir. order 
that the measures be obtained. 

In Step 3, we list the obstacles that impede the desired behaviors 
of Step l, and define tho»n obstacles in measurable term?. (PPS experience 
and logic are necessary to accomplish this and subsequent steps.) Pro- 
cedure© for monitoring change in such impeding behaviors should also be 
established at this stage. 

In Step 4, behaviors which will remove the obstacles in Step 3 arc 
listed and procedures are developed for continually monitoring the degree 
to which such behaviors occur. 

In Step 5, the behaviors which will prevent or curtail the Step 4 
behaviors are listed and we develop procedures for monitoring their oc- 
currence. 

In Step 6, methods of inducing the behaviors of Step 4 are defined 
in measurable terms, and arrangements for their measurement are made. 

Step 7 consists in defining procedures to minimize the behaviors 
of Step 5 which were thoughtto be counterproductive. 
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In Step 8 t the system is activated, starting with Step 7 and pro- 
ceeding to Step 1. The system is then reviewed to insure that feedback 
data are being provided. 

Step 9 consists in collecting data as it becomes available and analyz- 
ing their fit to our process and outcome criteria. Analysis should begin 
before data from all steps are available. Indeed, Step 9 must be ongoing 
with each new step in the system. Analysis of the immediate data will clar~ 
ify whether the program’s components are operating as specified and whether 
their inputs are producing the desired results. The plan of data analy- 
sis must Influence, but not dictate, the type and form of data collected 
at each step. Statistical knowledge is essential in planning and analy- 
sis, in order to maximize the data’s usefulness. 

In Step 10, all analyses should be summarized and recommendations for 
improvement should be made. Some improvements in the system will have 
already been implemented because of the ongoing nature of Step 9. Other 
improvements, however, may be possible only from hindsight. (All such 
improvements and recommendations will again require expertise in PPS.) 

After the first cycle, all recommendations should be Implemented and the 
system recycled. 

An Illustration of the evaluation schedule's application to a drop- 
out prevention program is presented in Figure 1. The type of behaviors 
and degree of specification needed for each step in the schedule are in- 
dicated for the hypothetical program. The content of Table 1 is abbre- 
viated for purposes of illustration and does not represent an operational 
dropout prevention program. 
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Insert Table 1 about here. 



Inspection of Table 1 Illustrates that there are several points In 
the process where data are provided which allow the potential effective- 
ness of the program to be judged before our product completes the program. 
Since our product is a human being, it is essential that we maximize the 
success potential of our program. Use of the evaluation schedule presented 
here permits us to do that. For example, in the dropout prevention pro- 
gram, Table 1, a PPS coordinator can quickly know whether his special pro- 
gramming and incentives have led students to take remedial reading and 
counseling. If they were not working, he could immediately Introduce new 
strategies for involving the students in hi9 two components, or he could 
add or substitute other components for removing the obstacle behaviors of 
Step 3. Of course, knowing that the components of the system are operating 
and that they are performing according to expectation does not guarantee 
achievement of the goal, because more components may be needed; however, 
if ve know the system is not operating, of that components are malfunc- 
tioning, we know ve are not likely to achieve our goal. 
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TABLE 1 

EVALUATION COMPONENTS OF A D ROPOUT- PRI i VEKTIOK PROGR AM 



STEP 

l 



b. 



4. a. 



COMPONENT 



« T j 



Increase percentage of entering 10 
their high school graduation reqt i 
Semi-annually the computer prints c it each student r s progress 
toward graduation. This report ij; compiled by scanning each 

em. 



h graders who complete 
I reman ts. 



I: read high school materials 
i grade reading score on a 
cs are administered yearly. 



student's courses and grades In t 
Students dropout because they cenno 
-• that is, they cannot make a 9t 
standard reading test. These tes 
Students drop out because they feel school is irrelevant* They 
cut excessively and do not parti jipate in class when there. 
Teacher reports of cutting and lion -participation are fed 
into the computer every grading [period, and therein each 
student is analyzed for dropout! potential. 

Participation in a remedial -eadflig course equips a high schooler 
to read at the 9th grade level/ Weekly attendance figures 
and monthly content ratings arlv provided for such courses. 

Group counseling which focuses cli developing one's philosophy 



of life increases the relevam 
fled. Weekly attendance and t 
provided for such groups. 



e of school for the dlssatis* 
onthly content ratings are 
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TABLE 1 (cont'd) 

EVALUATION COMPONENTS OF A PRO POUT- PREVENT ION PROGRAM 

STEP COMPONENT 

5 a. A combination of part time work and extra-curricular acti- 
vities Unit ability profit from remedial reading, 
b. Not attending school reduces participation in group coun- 
seling. Each student's cutting rate for each grading 
period is Available via computer. 

6. a. Remedial reading it scheduled at ecceislble times and places. 

Remedial reading materials are credible to adolescents and 
their credibility is rechecked in a yearly survey. Defi- 
cient readers are Invited to pertlclpate in such classes. 
Others may join. 

b. There are sufficient counseling groups, and such groups meet 
at accessible times. This is verified by semi-annual sur- 
vey. Potential dropouts are invited to participate and 
others may join. 

7. a. Activity schedules of student* with a reading deficiency are 

scanned after every grade report. Counselors interview 
those whose schedules ate too heavy, 
b. Students with high cut rates are Invited to attend group 
counseling even though they cut other school activities. 
Attendance at group is confidential. 
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TABLE 1 (coit'd) 

EVALUATION COMPONENTS OF A D R OPOUT* E VENT ION PROGRAM 

STSP COMPONEN T 

8. Remedial reading and group counseling are begun. Director 

monitors reports from instructors end teachers* and co* 
ordinates with school programmer to insure other data 
are being assembled, 

9, Director immediately determines percentage of deficient readers 

and potential dropouts participating in his components. He 
continues to monitor data shoving whether reading and coun- 
seling are be<ng provided according to plan* whether remed- 
ial reading raises reading level, whether this reduces 
dropping out, etc. 

10, Director summarizes data from first group’s experience in 

going through the ays tea, and in concert with hia PPS 
team proposes modifications, vhich are immediately im- 
plemented for entering group. 
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